Real-time prediction and management of food product demand

ABSTRACT

A real-time buffer manager system that calculates optimal food buffer levels, for both completed products and product components, based on real-time counts of restaurant patrons throughout a restaurant&#39;s property and the estimated time for them to arrive at a food ordering station. The real-time buffer manager employs a computer vision system, running a series of 2D image processing techniques that detect and track vehicles and people in several camera views. Patron counts are fed from the computer vision system into a queuing model that estimates when each patron will arrive at an ordering station. Thus, instead of analyzing historical sales data, the buffer manager according to the present invention electronically performs direct measurement of probable future demand, and electronically predicts, in real-time, what the future food product demand will be in a predetermined time (e.g., 3-5 minutes) immediately following the direct measurement of the demand.

BACKGROUND

1. Field of the Invention

The present invention generally relates to the restaurant industry, and,more particularly, to a system and method of real-time electronicprediction and management of food product demand, especially inquick-service restaurant industry.

2. Description of Related Art

The quick-service (or fast food) restaurant industry's primary valueproposition is speed-of-service—i.e., how quickly the restaurant candeliver a complete meal after a customer has placed an order.Quick-service restaurant operations are built upon the concept ofpreparing a limited menu of food product before customers place theirorders. By preparing food ahead of time and keeping it warm in a holdingbuffer, restaurant employees can quickly grab food product from thebuffer, bag it, and hand it to a customer. This faster speed-of-serviceenables quick-service restaurants to serve many more customers duringbusy mealtimes than a traditional sit-down restaurant.

In order to efficiently serve the customers (e.g., during a “rushhour”), a restaurant manager must carefully manage the “bufferlevel”—i.e. the amount of each food product that employees make andstore in the holding buffer throughout the day. The restaurant managerneeds to ensure that sufficient food product is on hand to meetanticipated demand, but must also ensure that the food product doesn'tsit too long before being served, as food product quickly deterioratesafter being prepared. Ideally, restaurant managers would like to bufferexactly the right amount of food product to serve consumers over thenext few minutes. Unfortunately, it's difficult for managers to know howmany consumer orders they will receive over the next few minutes, sothey are forced to buffer “extra” product, just to be safe.

The buffer management problem is fundamentally a trade-off between thequality of the restaurant's food and the speed with which the restaurantcan serve customers. The management of this trade-off is one of thelargest drivers of the restaurant's profitability, as speed-of-servicedrives sales as long as quality is maintained. Despite its importance tothe industry, restaurant managers can do little more than make “educatedguesses” as they lack one critical piece of information—when will theconsumers arrive?

The restaurant industry's state-of-the art solutions are based ondata-mining historical sales information to predict future orderingpatterns. Each restaurant saves a multi-week history of its salesvolumes for each product. For example, sales data of the past 70 daysmay be stored and analyzed. The data-mining software then averagesproduct volumes for discrete time windows (called “day parts”)throughout the day—e.g., 15 or 30 minute windows are common, for eachday of the week. The day-parts printout might suggest, for example,keeping 2 bins of patties on hand from 11:00 to 11:30, and thenincreasing the buffer to 3 bins from 11:30 to 12:00.

In another approach to buffer management, a program makes thedetermination of when to buffer completed sandwiches based on the sametheory of analyzing historical sales data, except that, in thisapproach, the sales data for the sale in the last few minutes of popularcompleted sandwiches (e.g., cheeseburgers) is reviewed. Thereafter, theprogram calculates the number of completed cheeseburgers to buffer as afunction of the number of cheeseburgers sold within the last fewminutes. Thus, again, the food production decision is based uponhistorical information (i.e., sales that have already occurred).

The current approach is based on the underlying assumption that “futureproduct demand will be very similar to historic (or past) productdemand”. This assumption is relatively true when product demand isaveraged over large time windows, for example Over a day. Restaurantscan have relatively high confidence that they will sell roughly the samenumber of cheeseburgers today as they did yesterday—assuming that pricesdo not change.

However, the current approach does not allow restaurant resources to bemanaged on a real-time basis because the variability of the correlationbetween past and future demand events is too large. In other words,historic information does not allow restaurant managers to know withconfidence the demand that their restaurant will see over the nextseveral minutes; however, restaurant performance (speed and quality)would benefit significantly if product demand could be predictedaccurately within the secondary shelf life of the restaurant's foodproducts.

The current approach suffers because it infers future demand fromhistoric demand, rather than taking a direct measurement of futuredemand. The reliability of the inference (i.e., the correlation betweenthe inferred demand and the actual demand) depends upon the time windowunder consideration. The current approach works fairly well when demandis correlated across large time windows (e.g., predicting the number ofpatties that will be consumed across a day). The current approachbecomes progressively less accurate as the time window shrinks. Further,current “inventory management systems” are inadequate to the needs ofthe restaurant industry because they do not account for the-limitedsecondary shelf life of food products.

As a result, data mining of historical sales generates demand estimateswith relatively large ranges. For example, in one restaurant, datamining may result in a prediction that for peak periods of the day thekitchen staff should pre-produce 4-7 bins of burger patties. If each binholds 18 patties, the range of production is 72 to 126 patties. Suchproduction predictions may be accurate (i.e., the predictions may handlethe customer demand during the peak periods) only because the range(which is 126-72=54 patties) is so large. Unfortunately, large variancesleave the restaurant vulnerable to over-production and, in practice,provide little more than rough production guidelines.

The area of management of food processing and food productionfacilities, such as the preparation of frozen dinners, differssignificantly from the production problems in quick-service restaurantsbecause it is not a real-time management of food processing and foodproduction. Processing facilities schedule production and makeproduction decisions based upon sales forecasts that are weeks andmonths ahead. Production times and rates can vary far more than inquick-service restaurants, with little effect, as the output of thefacility is destined for warehousing and distribution facilities. Wherequick-service restaurants must make minute-by-minute decisions,production facilities can often afford to make daily, or even weeklyproduction decisions. Thus, the needs of production facilities andquick-service restaurants vary greatly. The decisions to be taken at afood processing facility are not impacted by the minute-by-minutechanges in the demand for that facility's products.

Therefore, it is desirable to improve a restaurant's demand predictionaccuracy so as to enable restaurant mangers to use their productionresources more efficiently—increasing same-store profitability byimproving speed-of-service, food product quality, and reducing foodproduct wastage. More accurate demand prediction enables restaurantoperations to:

(1) Avoid under-production because under-production slows therestaurant's speed-of-service. When the buffer runs out of a certainfood product, then customers must add the food production time (often afew minutes) to their wait time. This is especially problematic for aserial queue, like a drive-thru where every customer in line must addthe food production time to his or her wait time. Under-production canseriously damage the restaurant's profitability by reducing the numberof customers served during peak meal times.

(2) Avoid over-production because over-production reduces therestaurant's food quality and increases wastage, as food product spendstoo much time in the bin. If the food product's bin time exceeds thesecondary shelf life, then it must be wasted.

(3) Minimize bin time because minimizing bin times means hotter, fresherfood products—a well-known market share driver. Restaurants would preferto minimize the amount of time that food products spend in the buffer.

(4) Pre-produce completed products because accurately bufferingcompleted products can significantly drive the restaurant's sales byimproving the restaurant's speed-of-service. Restaurants can buffer bothfood product components (e.g., burger patties) and completed foodproducts (e.g., plain cheeseburgers); however, the secondary shelf lifeof a completed food product is much shorter (2-3 minutes) than that of afood product component (30 minutes).

(5) Reduce “wasteful” production when a restaurant attempts to buffercompleted food products based on the historical sales data approach.This method is open to a significant number of incorrect guesses, whichnot only waste the food product (e.g., unused sandwiches orcheeseburgers), but also consume valuable production time that wasallocated to making a product that no one used.

As discussed before, a fundamental limitation of the current approach isthat the analysis of historical data only infers a range of probablefuture demand. It is therefore further desirable to improve the accuracyof future demand prediction using a method that moves beyondinference—to a direct measurement of demand or leading indicators ofdemand. In other words, it is desirable to base future food productiondecisions on the number of customers currently on the restaurantproperty who have not yet placed orders (i.e., sales that have not yetoccurred, but are likely to occur within the next several minutes).Direct measurements of demand will enable restaurants to improve theeconomic efficiency of their real-time production operations—ensuringthat production resources are assigned to profit-enhancing tasks.

SUMMARY

In one embodiment, the present invention contemplates a method ofmanaging food supply in real time. The method comprises electronicallygenerating real time data about food consumers inside or in the vicinityof a food outlet; and electronically predicting, based on the real timedata, an amount of food to be ordered by the food consumers in apredetermined time interval immediately following the generation of thereal time data. The method also includes electronically performing atleast one of the following in real time: estimating impending foodproduct demand in view of the prediction of the amount of food to beordered by the food consumers; and estimating demand for each completedfood product available for consumer consumption.

In another embodiment, the present invention contemplates a method ofmanaging food production and delivery in real time. The method compriseselectronically predicting, based on real time data, an amount of food tobe ordered at a food outlet in a specified time interval immediatelysucceeding a generation of the real time data; preparing the amount offood predicted to be ordered; and serving prepared food to patrons ofthe food outlet.

In a still further embodiment, the present invention contemplates acomputer-readable data storage medium containing a program code, which,upon execution by a processor, causes the processor to perform thefollowing in real time: generate data about food consumers inside or inthe vicinity of a food outlet; and predict, based on the data, an amountof food to be ordered by the food consumers in a predetermined timeinterval immediately following the generation of the data.

A real-time buffer manager system according to the present inventioncalculates optimal food buffer levels, for both completed products andproduct components, based on real-time counts of restaurant patronsthroughout a restaurant's property. The real-time buffer manager employsa computer vision system, running a series of 2D image processingtechniques that detect and track vehicles and people in several cameraviews. The system's cameras may be pointed at any of several keylocations throughout the restaurant's property, including the property'sentrance and exits, the drive-thru lane, the parking area, therestaurant's entrance and exit doors, and the front counter area. Patroncounts are fed from the computer vision system into a queuing model thatestimates when each patron will arrive at an ordering station.Simultaneously, a parametric observer takes inputs from the kitchen crewto track several key pieces of production information including thenumber of products and components in the food buffer, and averageservice times for ordering and production. The buffer manager estimates,in real-time, the probable demand for completed food products and foodproduct components, based on the number of patrons detected and theestimated time for them to arrive at an ordering station. Thisinformation is displayed to the kitchen crew, who then can prepare therequired food products and food product components. Thus, instead ofanalyzing historical sales data, the buffer manager according to thepresent invention electronically performs direct measurement of probablefuture demand, and electronically predicts, in real-time, what thefuture food product demand will be in a predetermined time (e.g., 3-5minutes) immediately following the direct measurement of the demand. Thebuffer manager may also manage restaurant employees.

BRIEF DESCRIPTION OF THE DRAWINGS

The accompanying drawings, which are included to provide a furtherunderstanding of the invention and are incorporated in and constitute apart of this specification, illustrate embodiments of the invention thattogether with the description serve to explain the principles of theinvention. In the drawings:

FIG. 1 illustrates a general flowchart depicting steps involved inoperating a quick-service restaurant according to one embodiment of thepresent invention;

FIG. 2 shows an exemplary food buffer manager system according to oneembodiment of the present invention;

FIG. 3 is a depiction of a real life implementation of the buffermanager system in FIG. 2 in a quick-service restaurant;

FIG. 4 is a detailed flowchart showing exemplary steps involved inmeasurement and prediction of impending food demand in real-timeaccording to one embodiment of the present invention;

FIG. 5 shows various detector units employed by an exemplary buffermanager as per the teachings of the present invention;

FIG. 6 illustrates an exemplary entrance detector architecture;

FIG. 7 shows an exemplary drive-thru detector architecture; and

FIG. 8 individually depicts various functional elements constituting thebuffer manager system according to a preferred embodiment of the presentinvention.

DETAILED DESCRIPTION

Reference will now be made in detail to the preferred embodiments of thepresent invention, examples of which are illustrated in the accompanyingdrawings. It is to be understood that the figures and descriptions ofthe present invention included herein illustrate and describe elementsthat are of particular relevance to the present invention, whileeliminating, for the purpose of clarity, other elements found in typicalquick-service (or fast food) restaurants.

It is worthy to note that any reference in the specification to “oneembodiment” or “an embodiment” means that a particular feature,structure or characteristic described in connection with the embodimentis included in at least one embodiment of the invention. The appearancesof the phrase “in one embodiment” at various places in the specificationdo not necessarily all refer to the same embodiment. It is further notedthat although the discussion below refers to a quick-service or fastfood restaurant, the discussion equally applies to any restaurant orfood outlet and is not limited to a fast food restaurant setting.Further, the term “food” may include a “solid” (e.g., burger, fries,sandwiches, etc.) as well as a liquid (e.g., coffee, carbonated drinks,juice, etc.) edible product. Also, the term “buffer manager”, as usedhereinbelow, refers to a computer unit, computing software, or any otherelectronic device that is configured to electronically perform real-timeprediction of food demand by executing various steps as illustrated, forexample, in FIG. 4. The buffer manager may operate in conjunction withone or more sensors and other data input devices as discussed below withreference to FIG. 2. Such generally-available sensors or data inputdevices may or may not form part of the entire buffer management systemaccording to the present invention depending on the type ofimplementation as discussed with reference to FIGS. 2 and 3 below.

FIG. 1 illustrates a general flowchart depicting steps involved inoperating a quick-service restaurant according to one embodiment of thepresent invention. A food buffer manager according to one embodiment ofthe present invention electronically generates, in real-time, data aboutfood consumers inside or in the vicinity of the quick-service restaurant(block 10). A food buffer is the holding buffer in a quick-servicerestaurant where food that is prepared ahead of time is kept warm andfrom which restaurant employees can quickly grab the ordered foodproduct, bag it, and hand it to the customer. Generally, there are twokinds of food buffers in a quick-service restaurant: those that holdcompleted food products (typically called “bins”), and those that holdcooked food product components (typically called “cabinets” or“component buffers”). The restaurant employees can quickly fulfill anorder by custom making a sandwich from the food product components thatare stored in a cabinet or component buffer. The buffer managercalculates optimal buffer levels based on measurements of impendingdemand in real-time. The buffer manager may be distinguished from thetraditional methods, where buffer level calculations are not determinedfrom real-time demand measurements, but from historical demandmeasurements.

The demand that a quick-service restaurant will see over the next 3-5minutes can be quickly estimated manually by standing on the roof of therestaurant and watching the traffic flow through the property. It isassumed that people are coming to the restaurant to buy food. If it canbe known how much food the average person orders, then one can form agood estimate of how much food needs to be cooked ahead of time, simplyby counting cars. The buffer manager may calculate optimal bufferlevels, for both completed products and product components, based onreal-time counts of restaurant patrons throughout the restaurant'sproperty (block 10, FIG. 1). In one embodiment, as discussed in moredetail later, the buffer manager employs a computer vision system,running a series of 2D image processing techniques that detect and trackvehicles and people in several camera views. The system's cameras may bepointed at any of several key locations throughout the restaurantproperty, including the property's entrance and exits, the drive-thrulane, the parking area, the restaurant's entrance and exit doors, andthe front counter area. Thus, the buffer manager, when installed withappropriate sensors, monitors customer flow and estimates foodproduction needs in real-time.

Computer vision technology may be used to measure leading indicators ofimpending demand including, for example, the arrival and movement ofpeople and vehicles throughout the restaurant's property. The number ofburger patties and fries (for example) that the restaurant will need tocook can be quickly estimated from the number of cars entering theproperty. More accurate estimates of when these products will be orderedcan be made by noting how long it takes for patrons to reach an orderpanel. The buffer manager can also observe how many vehicles are in thedrive-thru and how quickly the drive-thru is moving to estimate whenfuture drive thru orders will be placed. The buffer manager can alsonote how many vehicles are parked, how long it takes for customers towalk from their vehicles to the counter, and how long it takes foremployees to take a customer order, to estimate when future counterorders will be placed. Thus, the demand indicators may also include, forexample, customer ordering pattern, drive-thru delay, waiting timebefore ordering, time taken by an employee to take an order, etc.

Patron counts are fed from the computer vision system into a queuingmodel (also a part of the buffer manager) that estimates when eachpatron will arrive at an ordering station. Simultaneously, a parametricobserver may take inputs from the kitchen crew to track several keypieces of production information including the number of products andcomponents in the buffer, and average service times for ordering andproduction. The buffer manager may then electronically estimate theprobable demand for completed food products and food product componentsas a function of time, based on the number of patrons detected and theestimated time for them to arrive at an ordering station as given atblock 12 in FIG. 1. The prediction of food orders using real time datamay then be displayed to the kitchen crew or sent as direct commands toappropriate automated food production equipment, who can prepare ormaintain the required food products and food product components (block14) for serving or selling to the restaurant patrons (block 16).

FIG. 2 shows an exemplary food buffer manager system 18 according to oneembodiment of the present invention. The system 18 may include acomputing unit 20 that receives and processes data received from anumber of sensors 22-1, 22-2 located/placed inside and in the vicinity(or surrounding) a quick-service restaurant building. Although only twosensors 22-1 and 22-2 are illustrated in FIG. 2, it is observed thatthere may be many more sensors feeding data (e.g., customer arrival,drive-thru car progress, etc.) to the computing unit 20 as illustrated,for example, in FIG. 3. The computing unit 20 may internally store(e.g., in the unit's hard drive or other permanent storagememory/facility within the unit) appropriate program code that, uponexecution by one or more processors in the computing unit 20, may allowthe computing unit 20 to perform the buffer manager functionality. Inone embodiment, the computing unit 20 is a Windows® based personalcomputer. However, a computer running on any other operating system(e.g., Linux®) may also be configured to perform the buffer managerfunctionality. The program code may be written in C, C⁺⁺, or in anyother compiled or interpreted language suitably selected. The necessarysoftware modules for the program code may be designed using standardsoftware tools including, for example, compilers, linkers, assemblers,loaders, bug tracking systems, memory debugging systems, etc.

On the other hand, as shown in FIG. 2, the program code may reside on anexternal computer-readable, tangible data storage medium 24 (e.g., acompact disc, an optical disc, a magnetic storage medium such as afloppy disk, etc.) allowing ease of software portability and systemmanagement. More updated version of the program code may then be easilyloaded onto the internal memory of the computer 20 to enable thecomputer 20 to perform more sophisticated and advanced data and imageprocessing operations.

Customer arrival measurement (as discussed later hereinbelow) may beperformed with any of a number of sensors 22 including mechanical,electrical, and chemical sensors. Mechanical sensors may include a widevariety of switches, such as piezo-electrics, pneumatic pressureswitches, etc. Electrical sensors may include magnetic loop sensors,laser beam switches, and cameras (e.g., video surveillance cameras).Chemical sensors may also be used to detect vehicle passage by measuringchanges in CO (carbon monoxide) emissions. A particular implementationof a single type of sensors or a combination of different types ofsensors 22 may be based upon the relative cost/benefit of the approach.Some other sensing methods and approaches include:

(1) Radiative Sensing—The application may require a sensor capable ofdetecting vehicles and people through the restaurant's property.Radiative sensors such as cameras, lasers, or sonar may be used to sensesuch objects in a relatively large field of view and at a distance. Theability to detect an object far from an ordering panel, in order toprovide the kitchen with sufficient warning, especially recommends theuse of radiative sensing. In one embodiment, the radiative sensing isused to provide a 2D (two dimensional) image of the field of view ofinterest. Alternative sensors may also be used to extract 3D (threedimensional) information from the same general field of view. The use ofmultiple cameras, for instance, may be used in conjunction withstereovision techniques to generate a 3D model of an object in the fieldof view. Similarly, sonar sensors may be used with certain 3D processingtechniques to generate 3D occupancy models for a given field of view.These and other techniques may be employed to construct a 3D image ofthe object in the field of view.

(2) Contact or Field Sensors—A general alternative to radiative sensingis to employ some type of contact or field sensor, such as a limitswitch, a pressure switch or a magnetic loop sensor. For example, amechanical pressure switch may be laid across the restaurant entrance,generating a “click” each time a car drove across the switch. However,there may not be desirable cost/performance for a pressure switch. Theremay be multiple problems with these sensors in practice, including: (i)Pressure sensors may actually cost more than a camera; (ii) Multiplesensors may be required to determine whether a vehicle is entering orexiting; (iii) Vehicles may only be detected and counted while inmotion—a problem in the drive-thru where a vehicle may not move for sometime; (iv) Error propagation grows as the sum of misinterpreted clicks;and (v) Difficult to generalize techniques to also detect people (asopposed to just detecting the vehicles). Thus, generally, although anacceptably accurate vehicle detection system can be built using contactor field sensors, the cost in practice may be too high for theachievable performance and reliability.

It is noted here that as it is not difficult to reliably detect vehiclesin a parking lot of a restaurant when one knows roughly where to look,and what to look for, the designer of the buffer management systemaccording to the teachings of the present invention may have a wideselection of available sensing techniques from which to develop anacceptably accurate vehicle (and people) detection system.

As noted before, the term “buffer manager” may thus include the computer20 with the program code stored therein to allow it to perform thebuffer manager functionality according to the present invention (asillustrated, for example, in FIG. 4). However, the entire system 18 maybe considered a “buffer manager” if, for example, a restaurant operatorwishes to install sensors 22 and the computer 20 together from a singlevendor instead of obtaining the computer unit 20 from one vendor and thesensors 22 from another. On the other hand, instead of a single computer20, there may be more than one computing or processing units who jointlyperform the buffer manager functionality as is done in a preferredembodiment discussed immediately below. In other words, the buffermanager functionality may be distributed among two or more processingunits connected to one another via a network (e.g., the Ethernet)connection. Thus, the term “buffer manager” is used flexibly throughoutthe discussion and its meaning should be evident from the context ofuse.

In a preferred embodiment, the sensors 22 for drive-thru and customerarrival areas are Vision-Tech black and white, weatherproof, standardsecurity cameras with each camera rated at 0.5 lux and 380 lineresolution. The functionality of the computer 20 is implemented viathree hardware components—all connected to one another via an Ethernetconnection. The first hardware component is a demand prediction systemthat receives signals sent by the cameras. The first hardware componentis a Shuttle SS50C computing system with Intel P4 (2 GHz) processor, 512MB RAM, 20 GB hard disk space, and two (2) Winnov 1020 framegrabbers.The second hardware component is a manager console, which is a ShuttleSS50C system with Intel P4 (2 GHz) processor, 256 MB RAM, 20 GB harddisk space, and a 10 Base 2 network card. The third hardware componentis a kitchen production system, which is a Dell GX300 computer systemwith Intel P3 (800 MHz) processor, 256 MB RAM, 10 GB hard disk space,built-in network card, two (2) integrated serial ports, integratedvideo, ATI video card (PCI), STB video card (PCI), and two (2) serialport expansion slots (PCI). The display of various information to therestaurant employees and data input from them into the buffer managersystem are achieved via a number of serial-touch display terminals orinterfaces. The display terminals or interfaces may be placed at anumber of locations in the restaurant. In one embodiment, one displayinterface is placed at the grill area in the kitchen, the other isplaced at the frying area in the kitchen, and a third one is placed atthe food assembly area. Each display terminal or interface is a DataluxLMC 10 LCD touchscreen monitor with 640×480 resolution, 200 nitbrightness, and 250:1 contrast.

FIG. 3 is a depiction of a real life implementation of the buffermanager system in FIG. 2 in a quick-service restaurant 26. Therestaurant 26 is schematically depicted to have a drive-thru section 28,an eat-in/ordering area 30, and a front counter/order-taking area 32,which may include the food preparation/kitchen area (not shownseparately). The computer unit 20 (also shown in FIG. 2) that isconfigured to perform the buffer manager functionality using thesoftware or program code developed to implement the teachings of thepresent invention is shown placed in the counter area 32. The drive-thrusection 28 is shown to have two vehicles 50, 52 in the ordering lane.The diagram in FIG. 3 also shows the entry and exit areas on therestaurant property. A couple of other vehicles (or cars) 54, 56 arealso shown parked in the parking space provided for customers who wishto enter the restaurant and order and eat-in food there. It is observedthat the diagram in FIG. 3 is just a schematic representation of a fastfood restaurant or outlet, with two representative customers 60, 62shown in FIG. 3. In reality, there may be many more cars or othervehicles on the restaurant property, there may be many customers insidethe restaurant (i.e., in the eat-in area 30), or there may be more orless sensors placed at the restaurant than those shown in FIG. 3.

As discussed before, various types of sensors may be placed in and/oraround the restaurant property. In FIG. 3, one such implementation isshown with four sensors 34, 36, 38, 40 located on the four outside wallsof the restaurant eat-in establishment to detect and track vehicles andhuman traffic throughout the property, including the property's entranceand exits, the drive-thru lane, the parking area, the entrance and exitdoors (not shown) of the eat-in area 30, etc. Similarly, four othersensors 42,44, 46, and 48 are shown placed at different angularlocations within the eat-in area 30 to monitor the front counter area 32and also to monitor, for example, the number of patrons inside theestablishment, the ordering pattern of the customers inside therestaurant building, how long it takes for a restaurant employee to takea customer's order, etc. As noted before and discussed in more detaillater hereinbelow, various parameters tracked and measured by thesensors are sent to the buffer manager 20 in real-time, which analyzesthe sensor output data so as to estimate or predict, in real-time, thefood product demand that the restaurant may experience in apredetermined time (e.g., next 3-5 minutes) immediately following thesensor measurements.

It is observed here that although FIG. 3 shows the buffer manager 20 tobe an integral part of the restaurant 26, that may not have to benecessarily so. For example, the buffer manager 20 may be physicallylocated at a remote location (e.g., a central headquarter of therestaurant franchisor, or at another restaurant in a chain ofrestaurants under common ownership) and only a display monitor orcomputer terminal may be provided in the counter area 32 to guide andinstruct the kitchen staff. In such a setup, the remotely located buffermanager 20 may be linked or connected to one or more “client”restaurants via one of many available high-speed data connectionoptions, including, for example, cable-modems, WLL (wireless local loop)or any other wireless network, the Internet or any other similar datacommunication network, etc. The monitored data output from the sensorslocated at “client” restaurant may be sent over the high-speedconnection to the remotely located buffer manager, which, in turn, mayinterpret (or process) the data supplied by the specific group ofsensors at a specific restaurant and send back necessary food productionand maintenance information (i.e., its prediction of impeding foodorders) to that specific restaurant to guide the kitchen staff in foodpreparation. The operations of a buffer manager computer may be remotelymonitored (preferably periodically) by another computer (not shown) todiagnose and prevent any malfunction or fault condition occurring at thebuffer manager level. In one embodiment, food orders may be taken over anetwork (e.g., the Internet). In another embodiment, a food orderdelivery notification may be sent to a consumer over the network and theconsumer, in turn, may pick up the ordered food at a designatedlocation.

FIG. 4 is a detailed flowchart showing exemplary steps involved inmeasurement and prediction of impending food demand in real-timeaccording to one embodiment of the present invention. It is noted thatalthough the steps in FIG. 4 are illustrated in a sequential order, itis clear from the discussion given hereinbelow that one or more of thesteps in FIG. 4 can be performed in a order that is different from thatshown in FIG. 4. For example, the steps at blocks 84 and 88 in FIG. 4can be performed simultaneously, i.e., these steps do not have to beperformed sequentially. It is observed that the steps outlined in FIG. 4are implemented using the buffer management system 18 shown in FIG. 2.In other words, upon receiving sensor outputs, the buffer manager 20processes the received data to electronically carry out all the steps(except step 70) illustrated in FIG. 4. The placement of sensors at aplurality of locations on or around the restaurant property (block 70)is already discussed hereinbefore with reference to FIG. 3. The sensorsmay comprise the sensing system 73, whose output is processed (inreal-time) by the image processing system 74 to measure, in real-time,the leading indicators of impending demand, including, for example, thearrival and movement of people and vehicles throughout the restaurantsproperty, waiting time, ordering pattern, drive-thru delay, etc. Thedata measured at block 72 is fed as an input to an appropriate queuingmodel (discussed later hereinbelow) to estimate time of arrival for eachpatron at the front counter/ordering panel. Each queuing model shares acommon set of model parameters—the arrival time and the service time.The arrival time indicates the time at which a new patron (person orvehicle) enters the queue (including the drive-thru queue). The servicetime indicates the amount of time spent servicing an individual patronprior to that patron exiting the queue. These model parameters may beestimated, either through direct measurement or the construction of aparametric observer.

The queuing models predict the time at which a customer will reach anorder panel based on the customer's arrival time, the number of othercustomers in the queue, and the queue's average service time. In oneembodiment, the buffer management system 18 uses cameras and 2D imageprocessing techniques to measure the arrival time of each customer.

Sensing System 73

Various types of sensors and their utilities are described hereinbefore.Although a preferred embodiment uses cameras and 2D image processingtechniques to measure the arrival time of each customer, however, asnoted before, the buffer management according to the present inventioncan be equally performed with any of a number of other sensor and dataprocessing techniques. Cameras and 2D image processing may be chosenprimarily due to cost constraints. 3D sensing techniques, albeit moreexpensive, may be preferable for higher performance results.Fundamentally, the arrival time measurement system provides for theaccurate detection and/or tracking of an object within a field ofinterest. The field of interest must be located far enough away from theordering panel (i.e., the front counter area 32 in FIG. 3, whichincludes one or more ordering windows for drive-thru customers) suchthat detection of an arriving object provides the restaurant withsufficient warning that the kitchen can take action before the customerarrives at the ordering panel.

A sensor's required look-ahead may be determined from the restaurant'susual performance characteristics. If the restaurant requires an averageservice time T_(b) to perform a task, such as cooking a batch of fries,or preparing a batch of burgers; and an average service time T_(a) toassemble a burger, then the sensor should be placed at leastS*(T_(a)+T_(b)) from the order panel. The parameter S, in this case,indicates the average speed of the consumer. Sensor placement is afunction of the restaurant's performance—higher performance restaurantsdo not require as much warning time to react to changes in consumerdemand.

Image Processing System 74

In one embodiment, the buffer manager 20 employs 2D image processing ofseveral camera images in order to detect customers throughout therestaurant's property. It is noted that each queuing model may require apotentially different set of customer arrival measurements. Thefollowing discusses several 2D image-processing algorithms that are usedfor object detection, localization, and/or tracking. In addition,image-filtering methods that improve the quality of the image-processingalgorithms are also discussed. It is again noted that there are severalalternative methods of implementing object detection. The methodsdescribed herein have been chosen for their cost/performance benefits incertain vehicle/people detection applications according to oneembodiment of the present invention.

FIG. 5 shows various detector units employed by an exemplary buffermanager as per the teachings of the present invention. The softwaremodule that performs people and/or vehicle detection in the imageprocessing system 74 may include one or more of the following detectorunits: (i) an entrance vehicle detector 92 that detects and countsvehicles entering and exiting the restaurant property via restaurant'smain entrance, (ii) a drive-thru vehicle detector 94 that detects andcounts the number of vehicles in the drive-thru lane, (iii) a parkinglot detector 96 that detects and counts the number of parked cars, andalso detects and counts the number of patrons emerging from the cars,and (iv) a restaurant lobby detector 98 that counts the number ofpatrons entering the restaurant lobby or eat-in area 30.

FIG. 6 illustrates an exemplary architecture for the entrance detectorunit 92, and FIG. 7 shows an exemplary architecture for the drive-thrudetector unit 94 illustrated in FIG. 5. Image differencing is initiallyperformed as indicated by blocks 100, 102 and 104 in FIGS. 6 and 7.

Arrival Detector Implementation

The detection of arrival of a vehicle (e.g., a car) may be implementedby analyzing a live stream of images supplied by, for example, anentrance camera, and then signaling a software event when a car hasentered the restaurant property. In one embodiment, the overall flow ofthis analysis is broken down into three major processing steps, labeledP_(n), as follows: (1) P₁: Every image frame is processed to determinewhether or not a car, or significant portion of a car, is present in theimage. In a preferred embodiment, the image differencing method(described immediately below) is used to implement this P₁ processingstep. (2) P₂: If two consecutive frames are deemed to contain a car,they are compared to determine the direction of motion of the car. Theresult of this comparison is a “vote” for one of the following fouroutcomes: (i) “A”—Arrival, i.e., the car is moving in a directionconsistent with entering the property. (ii) “D”—Departure. (iii)“S”—Stationary, i.e., the car doesn't appear to move between twosuccessive frames. (iv) “I”—Indeterminate, i.e., in some cases, theimage data doesn't allow for a conclusive determination of the car'sdirection. The implementation of detection of motion or direction of acar in a preferred embodiment is described below. (3) P₃: During asequence of more than two consecutive frames containing a car, the votesgenerated by P₂ for each frame transition are tallied-until a frame isfound that does not contain a car. At this time, a software event isgenerated containing the outcome of this voting process. The signalingof a detection event is described below.

Image Differencing—A Filtering Method

“Background subtraction” or “image differencing” (block 100) is an imagefiltering method that calculates the difference in pixel values betweentwo images. This filtering method is useful in the detection of objectssuch as automobiles in a fixed field of view. The output of imagedifferencing may be run through a threshold to generate a binary outputimage. In one embodiment, image differencing is used to filter imagestaken from the restaurant parking lot entrance and exit, and thedrive-thru lane, prior to applying algorithms that detect vehicles. Aninitial image (i.e., a background image 102) is captured from this fieldof view, which is known to be devoid of objects—in this case, cars. Alarge object such as a car should create a large change in many pixelsin subsequent images—making it fairly straightforward to infer thepresence of a car by applying simple detection algorithms to thedifferenced image.

Image differencing calculates, on a pixel-by-pixel basis, the difference(block 100) in pixel intensity between two images: the “current” image104, which is presumed to contain an object of interest; and the“background” or reference image 102. The intuition of the method is thatif the background can be subtracted out of an image, only the object ofinterest will remain. A limitation of the method may lie in theselection of a reference image that closely correlates to the backgroundcontained in the current image. The intuition here is that backgroundpixel intensities are constantly changing as lighting conditions change.In order to properly “subtract out” the background of the current image,a reference image whose pixel intensities are very close to thebackground in the current image is preferably required.

If the reference image is not closely correlated to the background ofthe current image, then when the reference image is subtracted from the“current” image, the difference between the pixel intensities of the“reference background” and the “current background” may remain. Thesebackground differences are unwanted “background noise”, which canobscure the object of interest, making it more difficult for subsequentalgorithms to detect the object of interest.

Image-Wide Threshold—A Single Car Detector

Certain camera images may have a field of view that is likely to containonly a single vehicle at a time. The entrance and exit camera may be setup to have such a view. Under such conditions, a simple detectionalgorithm such as an image-wide difference threshold (block 106) may beused to detect the presence of an object large enough to be a car. Theimage-wide difference threshold method calculates an image-wide value ofthe differenced image (at block 100) (for example, by summing theabsolute value of the image pixels) in order to measure the magnitude ofthe change between the background image and the current image. If theimage-wide intensity difference is large enough (when compared with apredetermined threshold), then it is determined that a fairly largeobject has moved into the view. The position of the object of interestmay be estimated by calculating the center of mass of the object in thedifferenced image (block 108).

In this method, the differenced image is analyzed to determine whetheror not a car is present. The analysis may include the following steps:(a) Every pixel in the difference image may be compared to a thresholdvalue to determine whether the pixel should be labeled as “occluded.”The result is an “occlusion map”, containing boolean values (true orfalse) at each pixel position, indicating whether the correspondingpixel is labeled occluded or not. (b) Each row in the occlusion map maybe then analyzed by counting the number of occluded pixels in the row.If this count exceeds a threshold percentage relative to the totalnumber pixels in the row, the row is labeled as “occluded”. (c) If thenumber of occluded rows in the difference image exceeds a thresholdpercentage relative to the total number of rows in the image, the imageis determined to contain an object large enough to represent a car.

Direction or Motion Detection

The differenced image may contain a “blob” of pixels (i.e., the mass ofthe object of interest) corresponding to the object of interest (e.g., acar), probably surrounded by spurious pixels (noise). The center ofthese pixels can be calculated by averaging their row and column values(block 108). Comparing successive images, given a sufficiently fastframe rate, may provide a simple method of determining the direction ofthe vehicle—which may be useful in determining whether the vehicle isentering or exiting the restaurant's parking lot (block 110).

The motion detection process references the intermediate results of theprocessing step P₁ (discussed hereinabove) for two successive framescontaining a car. Let F_(n-1), and F_(n) denote the first and second ofsuch two frames, respectively. In a preferred embodiment, the entrancecamera is oriented such that arriving cars move through the image fromtop to bottom. Image rows are numbered increasing from top to bottom ofthe image. The motion or direction detection process considers thetop-most and bottom-most occluded rows as determined during P₁. LetRT_(i) denote the top-most occluded row of frame F_(i), while RB_(i)denote the bottom-most occluded row of frame F_(i). There are fourclassification determinations: (1) If RT_(n-1)<RT_(n) ANDRB_(n-1)<RB_(n), the car's motion between frames F_(n-1) and F_(n) isclassified as “A” for arrival. (2) If RT_(n-1)>RT_(n) ANDRB_(n-1)>RB_(n), the car's motion between frames F_(n-1) and F_(n) isclassified as “D” for departure. (3) If RT_(N-1)=RT_(n) ANDRB_(n-1)=RB_(n), the car's motion between frames F_(n-1) and F_(n) isclassified as “S” for stationary. (4) In all other cases, the car'smotion between frames F_(n-1) and F_(n) is classified as “I” forindeterminate.

Signaling a Detection Event

An “event sequence” may be defined as a series of consecutive frameswhich contain a car. Thus, an event sequence begins when a framecontaining no car is followed by a frame that does contain a car;similarly, the event sequence ends when a frame containing a car isfollowed by'a frame not containing a car. All frames in between thesetwo endpoints are considered part of the event sequence. Everyconsecutive pair of frames in the event sequence is analyzed byprocessing step P₂ (discussed hereinabove) to determine its direction ofmotion, and the resulting votes are summed for the entire eventsequence. At the end of the event sequence, the motion indicator withthe highest number of votes may be used to classify the entire eventsequence as “A”, “D”, “S” or “I”, and a software event containing thisoutcome may be generated and broadcast for consumption by other softwaremodules. The vote counters are then reset to zero for the next eventsequence.

It is important to note that an image-wide thresholding algorithm mayprovide evidence that a certain metric of the object matches a knownmetric of the object class. In other words, this algorithm may say thatthe object is “large enough to be a car,” but may not positivelyidentify the object as a car. This algorithm can be “fooled” intogenerating a false positive by any other object that is also “largeenough to be a car”. For example, if a bus unloads a large group ofpatrons who happened to walk through the field of view, they could befalsely identified as a vehicle because they collectively create enoughchange in the image to register as “large enough to be a car”.

In addition, the image-wide thresholding method may not determine thenumber of cars, i.e., how many cars are in the field of view. This maycause several problems. For example, two cars may pass through the fieldof view in opposite directions at roughly the same time. In that event,the thresholding detector will indicate when something is in the fieldof view that's “large enough to be a car”, but will not indicate thattwo cars are present. Calculations to indicate the direction of the carmay also be in error, as the center of the “blobs” will correspond tothe average center of the two cars.

Improved Single and Multi Car Detection—Region Segmentation

Region segmentation algorithms can be used to overcome some of thelimitations of image-wide thresholding by determining whether the “blob”of pixels in the image represent a collected, compact mass. Spuriousnoise, and/or a collection of smaller objects (such as a group ofpeople), can fool the image-wide threshold by triggering as many pixelsas a car. Region segmentation techniques test whether those pixels areactually connected into a single blob—further evidence that the objectin the field of view is a car. In addition, region segmentationidentifies spurious events such as multiple cars passing through thefield of view together, even when they are passing in oppositedirections.

In one embodiment, images taken from several fields of view may containmultiple cars. This is especially important in tracking a drive-thrulane, where the objective is to maintain an accurate count of the numberof vehicles in the lane. Differenced images (at block 100) of theseviews may contain groups of pixels that correspond to the multiplevehicles. Therefore, it may be desirable to segment or group the pixelsthat comprise a single blob, in order that the number of distinct blobsin the image might be counted. Prior to segmenting, it may be desirableto perform pre-processing of the regions containing images by growingeach “blob” for corresponding images by 1-2 pixels in order to improvethe Connected Component Extraction algorithm's performance (block 112),as discussed below.

The “Connected Component Extraction” algorithm (block 114) is awell-known computer vision algorithm for labeling sets of pixels thatcomprise a single blob. The algorithm operates by marking pixels thatare “connected” (share a common edge and/or touch corners) with a commonlabel, thereby grouping the connected pixels into identified regions.The output of the algorithm is a list of regions. Each pixel in theimage may be marked with exactly one region label, to identify theregion to which it belongs. The number of vehicles in an imagecorresponds to the number of region labels. The location and thedirection of each vehicle's motion can now be calculated as in thesingle car example (blocks 108, 110).

A principal limitation of the region segmentation method arises from thenotion of “connection”. The algorithm, as discussed before, operates byscanning the image on a pixel-by-pixel basis and labeling connectedpixels with the same region label. In practice, image noise cansignificantly impact the performance of the algorithm by “disconnecting”regions that are actually connected in physical reality. Region growing(mentioned before with reference to block 112) is one method ofimproving the performance of region segmentation algorithms. Regiongrowing may be most easily applied to binary images by scanning eachpixel of the binary image. If the current pixel is occupied, then itsneighbors are checked—any unoccupied neighbor is set to an occupiedstatus in the output image. The name “region growing” comes from theobservation that the “blob” in the output image generally grows by onepixel around its perimeter.

Regions labeling methods may be improved by applying a threshold (block116) to the output regions to determine whether the region is (a) toosmall to correspond to a car, or (b) large enough to correspond to morethan one car. Regions that are too small may be eliminated from furtherconsideration. Regions that are large enough to correspond to more thanone car can be further processed, or simply noted to correspond to Ncars, where N is the ratio of the number of pixels in the region to thenumber of pixels corresponding to an average car. A ceiling or floorfunction may be applied to N to generate an integer output.

The accuracy of the 2-D image processing algorithms may be improved byaddressing changes in background lighting, which affect the quality ofdifferenced images; and noise in differenced images, which affects thequality of region segmentation algorithms. Background image drift can bemanaged by updating the reference image to correlate more closely withthe “background” contained in the current image. The reference image maybe updated on an image-wide, or pixel-by-pixel basis. Careful managementof the reference image helps to reduce or eliminate spurious “backgroundnoise” that inhibits the performance of subsequent image processingtechniques. A straightforward, image-wide method is to regularly selectthe most recent background image as the new reference image. In thismethod, the “current” image is differenced from the reference image. Ifthe differenced image is not found to contain an object, then thiscurrent image becomes the new reference image. A second, morecomputationally complex method is to maintain an average pixel value,for each pixel in the image, across the last several images. This methodessentially constructs an artificial reference image, rather thanselecting a new reference image. Careful selection of the averagingwindow enables the filter to react to abrupt changes in the background,without creating background image oscillation.

Feature Detection Approach to 2D Image Processing

As discussed above, one approach to 2D image processing is to focus on“region” or “blob” detection—i.e., to look for a group of pixels that islarge enough to be a car. An alternative image processing approach maybe to look for a set of “features” in the image that may provideevidence of the existence of an automobile or person in the image. Such“feature detection” techniques may operate by matching features in theimage to one or more feature models. In the case of an automobile, forexample, the “features” might be windshields, wheels, headlights, orother detectable features.

A feature detection method may benefit significantly from the use of“region” or “blob” detection methods as a pre-processing filter.Intuitively, feature detection methods “search” through an image,looking for a correlation with the feature model. The “region” or “blob”detection algorithms can significantly reduce the number of images to besearched—if there isn't anything large enough to be an automobile, itmay not be desirable to search that particular image for headlights. Ifregion methods do find something large enough to be an automobile, thenthe feature search can be focused on the region ofinterest—significantly improving the computation efficiency of thefeature search method. When used in conjunction with region or blobmethods, a feature detector may essentially test the hypothesis that the“object large enough to be a car” actually is a car by searching forcorroborating evidence—i.e., the existence of headlights, windshieldsand wheels.

Feature detection methods may also be used to differentiate betweenscenarios that may be ambiguous to region detection methods. Forexample, a region-based method may not be able to differentiate betweena pickup truck pulling a trailer through the drive-thru and two carsfollowing each other through the drive-thru. Both scenes may containblobs of roughly the same size and distance apart; however, a featuredetector may be able to confirm that, in the pickup and trailer scene(for example), the first blob is a vehicle, while the second is not.

Several key “features” may help to positively identify that an object isan automobile. The location and size of these features may also help toclassify the vehicle's type—i.e., for example, car vs. minivan vs. truckvs. bus. Some of the features may include: (1) Width—Most vehicles(i.e., vehicles that are more probable to visit a quick-servicerestaurant) may be considered to have the same general width. (2)Length—Vehicle lengths may vary, but may be considered generally to bebetween 15 and 22 feet. (3) Glass—Windshields and windows may be foundat 3-4 feet above the ground, and generally rectangular in form, andgenerally wrapped around the vehicle. (4) Wheels—Wheels and tires may besafely assumed to be round with wheel diameters of 13-18 inches.

Pixels corresponding to key features may be extracted from either anoriginal image or a differenced image. For performance improvement, asmentioned before, the image differencing technique may be used toidentify images likely to contain a vehicle—e.g., there is “somethinglarge enough to be a car” in the image. In the discussion under thissection, the term “original image” means the image that was originallydifferenced and found to contain an object. The term “differenced image”means the corresponding post-differencing output image.

There may be several edge operators commonly available—the simplest isprobably the Sobel operator and the most complex is probably the Cannyoperator. Edge operators may be chosen by the designer of the buffermanager 20 for a specific restaurant establishment by consideringparticular performance requirements of the desired application at therestaurant. The output of an edge detector may be an image of edgepixels. These edge pixels may then be examined to determine whether theylikely correspond to any particular vehicle features. In this section,it is assumed that an edge operator has been run over both the originaland the differenced images.

Running an edge operator over the differenced image may enable themeasurement of the basic width and length of the vehicle. Thisinformation may help to establish whether the blob (in the differencedimage) has the basic dimensions appropriate to a vehicle. Theinformation may further help to establish whether multiple vehicles arepresent in the image. This technique may be equivalent to measuring thesize of the blob, and identifying individual blobs using previouslydescribed region-based techniques.

Edge pixels may be grouped into geometric elements, such as lines andcircles, using standard 2D image processing techniques such as, forexample, the Hough Transform. The Hough Transform works by consideringthe possible set of lines to which each edge pixel might belong. Amatrix of the parameters of all possible lines, called the“accumulator”, may be formed. Each edge pixel may then add one vote tothe accumulator elements corresponding to lines of which it could be amember. Naturally, the accumulator elements corresponding to lines inthe image will receive many more votes—enabling their identification.The Hough Transform may be generalized to other geometric shapes. Forexample, circle detection may be achieved by using an accumulator thatreferences a circle's parameters (x,y,r) where x and y denote the centerof the circle and r is the circle's radius. Techniques other than theHough Transform may also be used either alone or in combination withother methods for feature matching.

Queuing Models

In one embodiment, the buffer manager 20 design is based upon classicalapproaches to uncertainty, queuing theory, parameter estimation, and 2Dimage processing (computer vision). Uncertainty is modeled by separatingthe uncertainty of buffer management into two orthogonal uncertaintyclasses and then addressing each class separately. These two uncertaintyclasses are: (i) Temporal Uncertainty—The uncertainty of when theconsumer order will be placed; and (ii) Product Uncertainty—Theuncertainty of what food product the consumer will order.

In one embodiment, temporal uncertainty (estimation of which isdiscussed in more detail hereinbelow with reference to block 82) may bereduced through direct measurements (i.e., the output generated at block72 in FIG. 4) of leading demand indicators. A quick-service restaurant'sbest leading demand indicator is the arrival of a customer on therestaurant's property. Future product orders are highly correlated tocustomer arrival, With a lag time that is a function of the number ofcustomers already on the property.

To estimate product uncertainty (discussed in more detail laterhereinbelow with reference to block 88), in one embodiment, eachconsumer order is modeled as an independent random event. Each productthen has a certain probability, p, of being selected by a consumerduring an order. The number of products expected to be consumed by Npatrons can be estimated as the summation of the outcomes of N randomevents.

Temporal and product uncertainty estimates may be combined to determinethe expected amount of product to be ordered within certain windows oftime. Conceptually, a future time can be assigned to each independentordering event, based on various measurements of that particularconsumer's progress through the restaurant property. The estimate ofwhen a consumer will place an order allows the calculation of theexpected demand for food products as a function of time.

In one embodiment, “Queue Theoretic Modeling” techniques are employed tobetter estimate when an arriving customer will place an order (i.e.,estimating order times). A queue theoretic model assumes that patronsarrive at a service location, where they form into a queue and areserviced according to some service rule. As an example, in a typicaldrive-thru restaurant queue patron vehicles line up in a single, serialqueue, where they are serviced according to a “First In, First Serve”rule. The selection of a queuing model may be a trade-off between demandprediction accuracy and computational complexity. The designer of abuffer manager for a specific restaurant may need to compare the queuingbehavior of that restaurant against a series of increasingly complexmodels to determine which model provides sufficient accuracy, given itscomputational cost, for that particular application. It may be criticalto improve the accuracy of the temporal component of the prediction bymore accurately modeling the queuing behavior of customers. Thefollowing is a series of queuing models listed in order of increasingcomputational complexity:

(1) Zero Queue Model (block 77)—The zero queue model may assume thateach patron arrives at the restaurant and is serviced in constantservice time, T. Note that, generally, service times may be replaced bya service time distribution with an average service time. Such techniquemay be useful in practice, as it can be used to establish “confidencebounds” on minimum and maximum expected service times. The zero queuemodel essentially assumes that service can be provided in an infinitelyparallel manner—in other words, the i^(th) patron's service time is nota function of the number of other patrons that are in the queue. Thus,conceptually, the zero-queue model treats the restaurant property as ablack box. If one were to track only when cars entered and exited therestaurant property, and calculate the average time that a car spends onthe property, then one would arrive at the average service time for thezero queue model. The model simply ignores the specifics of howcustomers are served within the restaurant property, forming only agross estimate of how long such service takes.

(2) Single Queue Model (block 78)—The single queue model assumes thateach patron arrives at the restaurant, enters a single queue with aconstant or average service time T. Conceptually, this model may beequivalent to a restaurant that provides only drive-thru service—veryvehicle enters the same drive-thru line, where the first person in lineis serviced.

(3) Single Queue, Multiple Station Model (block 79)—The single queuemodel can be expanded to multiple service stations, each of which has aservice time T. This model may be considered a conceptual expansion ofthe drive-thru-only restaurant, where service is provided at multiplestations. For example, drive-thru patrons often stop first at an orderstation, then proceed to a payment window, then proceed to a pick-upwindow.

(4) Multi Queue Model (block 80)—The multiple queue restaurant servicemodels are multiple, single queue models in parallel. Conceptually,patrons enter the restaurant property and then join one of severalpossible single queues. The most common example is the choice betweenjoining the drive-thru lane or parking the car and joining the counterqueue. Some restaurants also provide parallel counter service, wherepatrons can join one of several parallel counter queues.

Estimating Ordering Times—Queue Simulation The buffer manager 20 maycombine the visual detection of arriving consumers with a simulation ofthe restaurant's queue (using one or more of the queuing models 77-80)to predict when each consumer will arrive at an order panel (blocks 82,83).

A queue simulator may maintain a list of the objects detected by thevision system (i.e., the image processing system 74 operating on thedata received from the sensing system 73). Each object in the list mayhave an associated clock, which maintains the time since that object wasfirst detected. Each queue simulator, regardless of its complexity, maycreate a new object and add it to the list when the vision systemdetects a new vehicle entering the restaurant property at the entranceof the restaurant. The queue simulator may classify the state (orstatus) of each object. There are three possible states:transit-to-queue; in-queue; and served. The first state,“transit-to-queue”, denotes an object that is in transit to a queue, buthas not yet reached the queue. The second state, “in-queue” indicatesthat the object has arrived in a queue and is either waiting to beserved, or, if it is the first object in the queue, is currently beingserved. The third state, “served”, indicates that the object has beenserved.

Upon detecting a new object, the buffer manager 20 may add the newobject to the object list and start the object's clock. When a pre-settransit-time has elapsed (simulating the amount of time that it takesfor a detected object to enter the queue), the object's status may bechanged to “in-queue”. Once in the queue, the new object advances towardthe head of the queue, moving closer each time the object at the head ofthe queue is served. Once the new object reaches the head of the queue,a new clock may be started. Once a pre-set “service time” elapses, thenew object's status is changed to “served”.

In the case where an object enters a serial queue with more than oneservice station, additional states may be added to represent each stageof the serial queue. For example, a three-station queue may requireadding states to represent “first-stage-queue”, “second-stage-queue”,and “third-stage-queue”. Service clocks may also be required for eachstage. It is observed that when implementing a multi-stage serial queue,one must take-care to note the maximum number of objects that eachsegment of the queue will be allowed to contain. If an object's serviceis completed at the first service station, it may pass on to the secondqueue only if there is room. This requires modifying the test forchanging state to “if (service-time-has-elapsed) &&(room-in-the-next-queue)”. Parallel queues may be modeled as independentimplementations. However, there may still be a need to decide which ofthe parallel queues is entered by the object.

Implementation of a Zero Queue Simulator

This simulator models the restaurant as an infinite number of parallelservice stations. When an object is detected, a certain amount of time(transit-time) is assumed to elapse before the object reaches one ofthese service stations. Once at a service station, a certain amount oftime (service-time) is assumed to elapse to service the object. Azero-queue simulator may operate under the following state-change rules:(1) When a new object is added to the list: (a) Mark its status“transit-to-queue”; (b) Start the clock; (2) When T=transit-time haselapsed: (a) Change the object's state to “in-queue”; (b) Restart theclock; (3) When T=service-time has elapsed: (a) Change the object'sstate to “served”. A zero-queue simulator may require the estimation ofthe “transit-to-queue” time and the “service-time” parameters. Inpractice, however, the division between the “in-transit” and “in-queue”states is arbitrary and can be ignored. Thus, a single service time canbe estimated and used to model the average time that elapses from theentrance of a vehicle onto the restaurant property until that same carleaves.

Implementation of a Single Queue Simulator

This simulation models the restaurant as a single drive-thru. Everyobject detected at the entrance may be assumed to enter a single queue,which advances as the first object in that queue is served. Thissimulation models single point of service in the queue. A single-queuesimulator may operate under the following state-change rules: (1) When anew object is added to the list: (a) Mark its status “transit-to-queue”;(b) Start the clock; (2) When T=transit-time has elapsed: (a) Add theobject to the end of a serial queue; (b) Change the object's state to“in-queue”; (3) For the first object in the serial queue: (a) Restartthe clock; (b) When T=service-time has elapsed: (i) Change the object'sstate to “served”; (ii) Remove the object from the queue; (iii) Advanceeach object in the queue one position. A single-queue simulator mayrequire the estimation of a “transit-to-queue” time and a “service time”parameters for the single service station. An estimate of the“transit-to-queue” time can be formed by measuring the average amount oftime required for a car to transit from the entrance detector to the endof the queue. The “service time” can be estimated by measuring theaverage amount of time elapsed from the moment that a car pulls up tothe order panel until the time that it drives away from the present(delivery) window. These service times may or may not be available fromthe restaurant's ordering system.

Implementation of a Single Queue, Multiple Station Simulator

A single queue, multiple station model expands upon the single queuemodel by adding additional service points in the queue. The model isessentially a single drive-thru restaurant where service is divided intomultiple stations, such as order, payment, and presentation. A singlequeue, multiple station simulator may operate under the followingstate-change rules: (1) When a new object is added to the list: (a) Markits status “transit-to-queue”; (b) Start the clock; (2) WhenT=transit-time has elapsed: (a) Add the object to the end of a serialqueue; (b) Change the object's state to “in-queue”; (3) For the firstobject in the serial queue: (a) Restart the clock; (b) WhenT=service-time has elapsed: (i) If there is a next queue and room in thenext queue: (A) Change the object's state to “next-queue”; (B) Move theobject to the end of the “next-queue”; (C) Advance each object in thequeue one position; (ii) If this was the last queue, then: (A) Changethe object's state to “served”; (B) Remove the object from the queue;(C) Advance each object in the queue one position. A single queue,multiple station simulator may require the same transit-time estimatoras the single queue simulator. It differs from the previous model inthat it requires the estimation of multiple service times, one for eachstation in the queue. The estimation methods are essentially the same asin the single queue model.

Implementation of a Multi Queue Simulator

As discussed before, a multiple queue model is treated as multiple,single queue models in parallel. Hence, a multi queue simulator may beimplemented as multiple single queue simulators discussed hereinabove.Hence, no additional discussion of a multi queue simulator is providedherein.

Adding Drive-Thru Vehicle Detection

As described hereinbefore, 2-D image processing technique may be used tocount multiple vehicles in a single image. This technique can be appliedto a field of view that contains the drive-thru lane and used to countthe number of vehicles in the drive-thru. Such a technique may alsoenable the system to reason about expected counter traffic.

A direct measurement of the drive-thru queue can replace the need for adrive-thru queue simulator to model traffic flow through the drive-thrulane—assuming that the implementation of the vision processing systemprovides sufficiently accurate data. In one embodiment, the use of adrive-thru measurement system is envisioned primarily as a means ofaugmenting and not replacing the entrance detector. The use of adrive-thru measurement system alone may not provide the buffer manager20 with an accurate estimate of the total demand that the restaurantexpects to see in the next few minutes, as it may not have any means ofdetecting the number of vehicles that have entered the property, buthave not entered the drive-thru line.

In certain applications, where the drive-thru comprises a very largepercentage of the restaurant's business, the solo use of a drive-thrumeasurement system may prove sufficiently accurate. For theseapplications, a “safety factor” may be introduced to account for theundetected entrance traffic. Such implementations should be cautiouslytested to ensure that they truly provide sufficient buffer managementaccuracy.

A direct measurement of the drive-thru queue, together with an accuratecount of the number of vehicles that have recently entered therestaurant's property, may make it possible to determine the number ofvehicles that have been parked. Patrons of parked vehicles usuallybecome counter traffic, enabling the system to form an estimate of thearrival of new counter traffic that is substantially proportional to thenumber of newly parked cars. A counter queue may be modeled simply as asingle serial queue or a set of parallel queues, as appropriate to theactual ordering configuration of the restaurant front counter area 32(FIG. 3).

Implementation of a Simulator with Drive-Thru Visual Detection

Drive-thru visual detection may significantly reduce the uncertaintyassociated with the internal flow of consumer traffic. The actual stateof the drive-thru can be measured at a high enough rate to virtuallyeliminate uncertainty about the number of vehicles in the drive-thru andthe average service time for the queue. The queue simulator may be splitinto a drive-thru model and a counter model. Direct visual monitoring ofthe drive-thru lane enables the employment of the simple, single queuemodel without the need for multiple service stations, as discussedbelow. The counter queue may be modeled as a single queue, or a set ofparallel queues, depending upon the actual configuration of the counterservice area as noted before.

Direct visual measurements may enable the drive-thru simulator tooperate under some simple state change rules: (1) For each objectdetected at the entrance: (a) Create a new object; (b) Assign thatobject the status “in-transit”; (c) If an object enters the tail of thedrive thru queue within some “maximum-transit-time” then: (i) Change theobject's status to “drive-thru-queue”; (ii) Add the object to thedrive-thru queue list; (iii) Assign the object a position number—e.g.,i^(th) object in the queue; (d) Otherwise, assume that the object hasparked: (i) Change the object's status to “parked”; (ii) Add N personsto the counter traffic queue; (2) For each object in the drive-thruqueue: (a) Visually monitor the queue; (b) For the first object in thequeue: (i) Start a clock when it becomes the first object; (c) When thefirst object in the queue moves past the ordering panel: (i) Stop thefirst object's clock; (A) Report the elapsed time to the average servicetime estimator; (ii) Update each object's position in the queue; (iii)Change the first object's status to “served”.

A counter queue simulator may be implemented as either a single servicequeue, or multiple parallel service queues, depending upon the physicalservice layout in that particular restaurant. It is observed that manyrestaurants force their patrons to queue-up in a single line, usingrailings or other barriers to denote the queue path. Other restaurantsmay have multiple ordering stations, allowing patrons to line up behind2-3 different queues. A counter queue simulator's state change rules maybe identical to those of a single queue. Where parallel queues are used,the simulator may assign each incoming patron to one of the parallelqueues. In one embodiment, incidences of queue shifting are ignored.

The employment of visual monitoring of the drive-thru lane maysignificantly reduce the uncertainty of the queue simulation by reducingthe amount of traffic whose progress through the restaurant serviceprocess is simulated. With drive-thru traffic comprising 40%-60% of anaverage restaurant's daily business, this may result in significantreduction in prediction error. However, one must realize that theunderlying assumption in employing this method is that visual monitoringis more accurate than the drive-thru simulation. Therefore, one mustensure that this is indeed the case. Further, an implicit assumption ismade in visual monitoring that all traffic that does not enter the drivethru must have parked and become counter traffic. This may be unlikelyto be true—at least some traffic may pass through, may be employees, ormay not become counter traffic for some other reason. This assumptionmay be softened by introducing a proportionality constant—and, in fact,this implicitly happens, as such a constant is “folded in” to theconsumer to parked car ratio previously discussed.

Extension to Enable Detection of Parked Cars

The accuracy of the buffer manager 20 may be further improved by takingdirect visual measurements of a restaurant's parking area in order todetermine exactly how many cars have parked. Simple extensions to theimage processing algorithms previously described enable the system todetermine whether a vehicle has parked. Parked cars occupy regions ofthe parking lot that are known a priori and cars stay there for a fairlylong period of time. The detection of parked cars may be accomplishedusing substantially the same 2-D image processing methods previouslydescribed for tracking a drive-thru lane. Extensions to the imageprocessing methods may be required to calculate the positions of theidentified regions and determine whether they presently occupy a“parking space”. The positions of the identified regions can be reportedeither in the image space (column, row), or transformed into acoordinate system in the restaurant's frame of reference. In a preferredembodiment, the camera used to localize the object remains fixed in therestaurant's frame of reference—therefore transformation into a secondcoordinate system may not be necessary so long as the “parking spaces”can be transformed into the image space.

Occlusion may be a particular concern because the buffer manager 20 mayneed to distinguish between the occupancy of a parking space withocclusion of a parking space. There are two general approaches todistinguishing between occupancy and occlusion. The first method is toadd a second sensor whose field of view removes the ambiguity—i.e., thesecond sensor can see the object from another point of view, enabling a3D localization. The second method is to selectively place the originalsensor such that occlusions are minimized, and use a set of heuristicsto “confirm” that the object is occupying the space. Such heuristics mayinclude measuring the median axes of the object to see if the long axisaligns with the parking space.

Visual detection of parked cars may improve the accuracy of theestimates of the counter queue arrivals by more accurately measuring thetime at which a car parks. This method may still require the assumptionthat a certain number of consumers enters the counter queue per car,thus no improvement in the actual consumer count may be made. Asdiscussed before with reference to the state change rules for a singlequeue model, the single queue simulator created an object upon itsdetection at the entrance, and then assumed that the object spent someamount of time “in-transit”. The application of this queuing method tothe counter queue may require estimating the “in-transit” time toinclude the time to park the car, the time to get out of the vehicle,the time to walk across the parking lot, and the time to enter a queue.Visual detection of parked cars may improve the accuracy of thesimulator by actually measuring the first part of the “in-transit”time—the time to park the car.

Detecting People

The accuracy of a queue simulator may benefit from the visual detectionof people, either as they exit their parked cars or as they enter therestaurant. In one embodiment, the same cameras that provide images ofthe parking lot, also capture images of patrons as they exit theirvehicles. Additional cameras can be placed such that they capture viewsof the entrance doors at the restaurant. The addition of entrance doorcameras may provide higher accuracy patron count, as the field of viewcan be narrowed such that a person subsumes a larger number of pixelsthan is possible with a parking lot camera. The detection of a personmay be accomplished in substantially the same manner as the detection ofvehicles in the drive-thru—using image differencing, followed by regionsegmentation (as discussed hereinbefore with reference to FIG. 7).

The detection method may be applied to images of a parking lot in thecase where the detection of persons exiting their vehicles is desired.The structure of the problem enables the detector to narrow its searchfor new, emerging blobs that may appear to the right and left of thelong axis of the vehicle as the vehicle's doors open. This change in theshape of the vehicle is the first evidence that a person will mostprobably emerge from the vehicle. Detection of an opening door may beperformed by comparing iterative “blobs” and looking for the emergenceof additional pixels in the appropriate place. Following the detectionof the opening of the door, the buffer manager 20 may track that side ofthe vehicle (i.e., the side of the vehicle containing the opened door),toward the rear of the vehicle, for small blobs that may emerge from theparking zone, cross the parking lot, and head toward an entrance of therestaurant. Following region segmentation, the height and aspect ratioof blobs detected in the appropriate sections of the image may becompared to determine if they are the correct height and shape for ahuman. These metrics may further provide evidence as to whether the blobmay be an adult or a child.

The same method may also be employed to use a camera specificallyaligned with the entrance doorway. The addition of a sensor thatindicates when the door has been opened (such as a broken circuit) canprovide separate evidence that patron(s) may be entering. It is observedthat the application of the method discussed hereinabove to parking lotimages versus doorway entrance images may be a trade-off betweenlook-ahead and accuracy. The detection and count of patrons in theparking lot may provide the restaurant kitchen with more warning thatconsumers are approaching. The detection and count of patrons at theentrance doorway may be more accurate—especially if one attempts toclassify patrons into classes such as “adult” versus “child”—simplybecause a closer image will provide the vision algorithms with morepixels to work with. The visual detection of people entering arestaurant (whether detected exiting a car, or entering the doorway) mayprovide an explicit count for the counter queue. This explicit count maybe more accurate than an estimate based solely on the number of carsthat park in the restaurant's parking lot.

Estimating Impending Production Demand

After modeling temporal uncertainty (at block 82 in FIG. 4) to determinethe timing of consumer orders, the buffer manager 20 may next estimateimpending production demand (block 84) using a probabilistic model tocalculate the expected number of food products and food productcomponents the restaurant's kitchen should prepare in order to meet theimpending demand generated by arriving customers. The probabilisticmodel may take as input N, the number of patrons expected to arrive atan ordering panel within the time period T (as determined at block 82).Alternately, the probabilistic model may take as input N(t)—thetime-dependent version of the time-invariant input N—indicating thenumber of patrons expected to arrive at the ordering panel at time “t”(i.e., a particular instant of time as distinguished from an interval oftime given by “T”).

The buffer manager 20 may use queue simulation to make a series ofproduction decisions in real time. Some typical questions that a humanrestaurant manager may periodically consider in order to manage bufferlevels are: (1) How many bins of components should the kitchen crewgenerally keep in the buffer? (2) How much longer is the current binlevel likely to last? (3) When will the kitchen likely need to startcooking more food product components? (4) Can I confidently order anycompleted products to be made ahead of time?

The buffer manager 20 may estimate impending production demand byestimating desired nominal component buffer level (block 85), estimatingremaining time per bin (block 86), and estimating optimal preparationtimes (block 87) as discussed below.

Estimating Desired Nominal Component Buffer Level (Block 85)

A “nominal” buffer level may be defined as the number of bins of foodcomponents (e.g., burger patties and buns for a completed burgerproduct) that a human restaurant manager may order the kitchen staff tomaintain. In a restaurant with a grill, a grill cook can easily regulatethe buffer at a certain number of bins or patties. This method of buffermanagement is traditionally one of the most common because it is so easyfor the crew to accurately carry out. A human manager can simply tellthe grill cook to maintain 34 bins of meat throughout lunch and feelconfident that that level of food product will be available. In oneembodiment, the buffer manager 20 according to the present inventiondetermines the nominal buffer level using a patty count instead of a bincount.

The buffer manager 20 according to the present invention improves uponthis practice by electronically calculating a “nominal” buffer levelthat is proportional to the actual number of patrons on the restaurant'sproperty. The buffer manager 20 attempts to change the “nominal” bufferlevel such that both over and under production are avoided. Nominalcomponent buffer levels may be calculated based on the number ofpatrons, N, on the restaurant property who have not yet placed an order.As described hereinbefore, several vision-based methods of estimatingthe number of patrons on the restaurant's property include counting carsat the restaurant's entrance, counting cars at the drive-thru, countingparked cars, and counting restaurant patrons. From these severalmethods, an estimate of N may be formed. The nominal buffer level foreach food product component is proportional to N. The proportionalityconstant for each food product component may be termed the “expectedvalue”—referring to the expected number of food product components thateach patron is expected to consume, on average. Estimation of the“expected value” may be considered a part of the product uncertaintymodeling (discussed later hereinbelow with reference to block 88 in FIG.4).

Estimation of desired nominal component buffer level may boundproduction by the total number of unplaced orders on the restaurantproperty. Where historical data methods may actually call for productionduring periods where there are absolutely no potential consumers on theproperty, the estimation performed by the buffer manager 20 would callfor a bin level of zero. Where historical data methods may call for lowproduction during a period when 20 cars show up, the buffer manager 20would instruct the kitchen crew to drive production higher, inproportion to the newly arrived demand. Thus, this method of estimatingdesired nominal component buffer level according to the presentinvention may drive production in proportion to current demand.Subsequent methods may work to improve the convergence of this method'sdemand-curve following performance.

The method that estimates the nominal component buffer level may be astrict proportional controller that commands a buffer level proportionalto the total number of unplaced orders on the restaurant property. Thismethod may not take time into account. Thus, product freshness may notbe represented in the calculation of production commands, nor may theminimum time requirements of production be considered. These limitationsmay cause the buffer manager 20 to order production that producesproduct unnecessarily soon, or it may allow bin levels or patty countsto drop so low that the kitchen staff could not adequately prepareproduct fast enough to serve the next arriving set of consumers. Inother words, the above-described method of estimating nominal componentbuffer level may not consider the impact on freshness or the need forsafety margins.

Estimating Remaining Time Per Bin (Block 86)

Each bin of food product components may last a certain amount of time,dependent upon the rate of arrival of new consumers. Thus, it may bedesirable to estimate the remaining time for a bin of food productcomponents so as to decide when to begin cooking another bin of foodproduct components. If there isn't enough demand to consume theremaining bins, then the restaurant shouldn't prepare additional foodproduct. If there is enough demand to consume the remaining bins, thenthe kitchen may wish to wait to prepare the new food product components“just in time” in order to maximize its freshness.

When an estimate of the number of patties (an example of a food productcomponent) remaining in the buffer is available (using the methoddescribed hereinbelow with reference to block 88 in FIG. 4), theremaining time per bin may be estimated as follows: (1) Calculate thenumber of patrons, P that can be served, given the number of pattiescurrently contained in a bin. (2) Obtain an estimate of the number ofpatrons on the property P*. (3) If P<P*, then no additional patties arerequired: (a) Remaining time per bin is not defined. (4) If P>P*, then:(a) Using the relevant queue simulator model, walk backwards through thequeue to the Pth patron; (b) Record the estimated time of arrival (ETA)for the Pth patron; (c) The ETA of the Pth patron is the estimatedremaining time for the bin.

The method described hereinabove may be extended to multiple bins. Mostbuffer systems contain multiple warming bins, therefore it is desirableto extend the method to calculate the remaining time per bin for eachbin. The bins are assumed to be labeled i=(1, 2, . . . N) in order oftheir production—bin 1 was made first, followed by bin 2, etc., throughbin N. The extended method can be given as: (1) Calculate the totalnumber of patrons P_(i) that can be served, given the number of pattiesin the i^(th) bin; (2) Obtain an estimate of the total number of patronson the property, P*; (3) For each bin: (a) If sum[P₀ . . . P_(i)]<P*,then (i) The remaining bin time for the i^(th) bin is undefined; (b) Ifsum[P₀ . . . P_(i)]>P*, then (i) Using the relevant queue simulatormodel, walk backward through the queue to the P_(i) ^(th) patron; (ii)Record the estimated time of arrival of the P_(i) ^(th) patron; (iii)The ETA of the P_(i) ^(th) patron is the estimated remaining time forthe i^(th) bin.

Estimating Optimal Preparation Times (Block 87)

The previously described method of calculating the “nominal” bin levels(at block 85) may not explicitly calculate when another bin of foodproduct components should be prepared. The method at block 85 may beused to regulate buffer levels by displaying the nominal bin level tothe kitchen crew and relying upon them to judge when to prepare aproduct. Alternatively, the buffer manager 20 may estimate optimalpreparation times for the kitchen crew by continually reviewing thecomponent reserve held in a buffer with the current demand detected onthe restaurant's property. Moreover, the queue simulators (discussedhereinbefore) may provide a good estimate of the rate at which thatdemand will place future orders. This information may collectivelyenable the buffer manager 20 to calculate optimal preparation times,given an estimate of the production time required to prepare another binof food product.

A method for determining the “nominal” bin levels—or the total number ofbins required to meet the demand currently on the restaurant propertyhas been described hereinbefore with reference to block 85 in FIG. 4.The following is a method to calculate the production time for each bin.First, the buffer manager 20 may subtract the number of bins currentlyavailable in the kitchen service cabinet from the total number of binsrequired. The difference may be called the “total bin requirement”.Next, the buffer manager 20 may determine when the cabinet will likelyrun out of its current supply—the “buffer horizon”. A method ofdetermining the estimated remaining time per bin has been describedhereinbefore with reference to block 86 in FIG. 4. Next, the buffermanager 20 may determine when the first consumer will reach an orderingpanel after the buffer runs out of its current supply. The time at whichthis would occur may be called the “downtime horizon”. Finally, thebuffer manager 20 may estimate how long the kitchen crew may take toproduce a bin once they are ordered to do so—the “production horizon”. Amethod of forming such an estimate is described later hereinbelow.

The optimal production time (T_(i)) for the i^(th) bin may be calculatedas follows: (1) For each bin to be produced [1, 2, 3 . . .total-bin-requirement]: (a) Simulate adding (i-1) bins to the buffer;(b) Calculate the downtime-horizon for the simulated buffer; (c)Ti=downtime-horizon—production-horizon. Optimally, a restaurant shouldproduce a new bin at the last possible moment, such that it is ready forconsumption just when it is needed. Such “just in time” production mayminimize the component's bin time, providing for hotter, fresherproduct. However, in practice, the restaurant may choose to operate witha safety margin, keeping a few extra patties in the bin or maintain apredetermined minimum number of patties in the bin to guard against aproduction glitch. A temporal safety margin can be added to (1)(c)above, to provide room for error. A restaurant may produce patties byissuing a new cook command when T_(i) drops to zero (or near zero, givena safety margin).

Estimating Demand for a Completed Product

Impending completed product demand may be estimated (i.e., productuncertainty modeling at block 88, FIG. 4) from consumer volume usingmethods similar to those employed to manage a component buffer. Adistinction between component and completed product buffering is volume.In general, far fewer completed products are buffered than productcomponents because many consumers require a special order and secondaryshelf life for completed products is considerably shorter than that forfood product components (e.g., burger patties).

Most quick-service restaurants have one or two sandwiches or hamburgersthat dominate sales volume—and generally, one or two favored“preparations” for these sandwiches. For example, regular and plaincheeseburgers are likely to dominate many quick-servicerestaurants'sales volumes. Thus, an intuition behind this method ofestimating demand for a completed product is that, if a human restaurantmanager were able to closely monitor the number of patrons on therestaurant property, he or she would likely be able to predict that, “Ifthere are 10 people out there, 2 of them are going to order a regularcheeseburger”. In other words, the restaurant manager is intuitivelycalculating that this popular item will probably subsume 20% of thefuture orders for this group of consumers. Moreover, the manager'sconfidence is relatively high because there are 10 people out there—astatistically relevant sampling for this application. The methoddescribed herein seeks to automate this intuitive process by predictingwhen sufficient demand is present to justify confidence in deciding thatit is time to pre-produce a specific completed sandwich.

An “expected value” or the “expected number of completed food products”that may be required over a predetermined time period may be found bymultiplying the “expected value per order” by N (the number of unplacedorders). Several queue simulation methods have been describedhereinbefore to estimate the number of unplaced orders on the restaurantproperty. A method of estimating the “expected value per order” for eachcompleted food product is described later hereinbelow.

The “expected value” for a completed food product is the number of foodproducts that the buffer manager 20 expects that N unplaced orders willrequire, based on the restaurant's then-current product mix. A humanrestaurant manager may want to order the kitchen crew to produce witheither more or less confidence than the buffer manager 20, thus a“confidence filter” may be applied to the “expected value.” For example,in one embodiment, the expected value may simply be multiplied by again, to produce a modified expected value. The gain itself may be afunction of a second measure of the confidence that the completedproduct will actually be consumed within its secondary shelf life. Thecalculation of the mean time between orders (MTBO) for that particularproduct may serve as a secondary confidence measurement. The MTBO valuemay be scaled to produce an appropriate gain.

Further, the classification of automobiles (e.g., mini-vans vs. pickuptrucks) and/or people (e.g., adult, child, male, female) may be used tomodify the expected value for that particular order. Classificationtechniques can be used to improve completed product estimates for singleorders. Intuitively, a restaurant manager may be able to guess thatvehicles likely to contain children will have a higher probability ofordering children's meals, which contain, for example, single hamburgersor chicken nuggets. Similarly, restaurant managers may notice thatconstruction vehicles are more likely to contain occupants who willorder larger, meatier sandwiches, such as a burger with two or morepatties. The classification of vehicles into vehicle types may be usedto improve the expected value for that particular order. Thus, theidentification of a vehicle as a “mini-van” would enable the buffermanager 20 to increase the probable consumption of hamburgers andnuggets, and decrease the probable consumption of large sandwiches.Similarly, the classification of people as adult, child, male, female,etc. can be used to narrow the expected consumption for particularcompleted food products—and by extension, food product components.

Expected Value Per Order for a Single Product

As discussed hereinbefore, a queue simulator (i.e., the queue simulatoror simulators selected for implementation as part of the buffer managerprogram code) outputs the expected time for each consumer order. Thefollowing describes a method for estimating the number of products, of asingle type (e.g., cheeseburgers, fries, onion rings, etc.), that may beordered in a single consumer order. In other words, the method describedbelow attempts to answer the question: For each consumer order placed,how many cheeseburgers (as an example of a type of food product) doesthe human restaurant manager expect to be ordered through each suchconsumer order?

The expected value per order may be calculated by filtering therestaurant's order data over a specific time window. In one embodiment,filtering generally means an averaging, though weighted averaging may bemore appropriate. Sales of individual products may be recorded andaveraged using any standard or batch averaging technique. It is notedthat products should preferably be tracked according to their completedstate. Thus, for example, plain cheeseburgers may be tracked separatelyfrom cheeseburgers with ketchup only. Further, weighting factors may beincluded to skew the average toward more recent sales. Other filteringtechniques may be used to average out (or eliminate) anomalous events,such as the sudden sale of a very large number of products—for instance,the sudden sale of 25 cheeseburgers. These and other filteringtechniques may be necessary to maintain smooth, convergent behavior ofthe expected value of an individual product. Filtered individual productsales may be normalized by the number of orders placed or normalized bythe number of vehicles detected to generate the expected value per orderfor that individual product.

Buffer State Maintenance

The current state of a buffer may be maintained by collecting data fromthe kitchen employees (input block 90, FIG. 4) and updating a bufferstate database that may be maintained internally (e.g., stored on thecomputer 20 running the buffer manager program code) by the buffermanager 20. The buffer management system 18 may use multiple touchscreen monitors (not shown as part of the system 18 illustrated in FIG.2) to collect production data from, for example, the grill, frymachines, and food product assembly area. The data may also be collectedfrom the point-of-sale system. Generally, the production data (at block90) may cover the production of every food product component and itsaddition to the buffer; the production of a completed product; and thewaste of components and/or completed products. The removal of productcomponents from the buffer may be calculated from a usage menuassociated with the completed product. Thus, as an example, for everydouble cheeseburger made, two burger patties may be removed from thecomponent buffer.

Estimating Average Service Times

Queuing theoretic models require the estimation of various service timesat stations throughout the queue where services are performed prior tothe object moving on to the next station. Average service times may beestimated by taking direct measurements of the service processes orthrough the use of a standard parametric observer. In one embodiment,service times may be estimated from direct measurements at eachrestaurant. For example, a rough estimate of the service time may betaken with a stopwatch. It is observed that the average service timesmay not tend to vary considerably, thus estimates may be stable for longperiods of time.

For those applications where service times do vary over time, directmeasurement of the various events may be taken using, for example, thevision system. For example, by time-stamping the progress of an objectfrom entrance, to parking, or into the drive-thru lane, variousin-transit times may be estimated. Drive-thru and counter service timesmay be estimated either by using the vision system to mark the forwardprogress of the queues, or by time-stamping consumer order data andnoting the time elapsed between orders.

More sophisticated methods, such as the construction of a formalparametric observer to optimally estimate each of the average servicetimes by filtering together the several measurements previously notedmay also be implemented. These methods are well known. It is, however,noted that the use of such sophisticated methods may not produceappreciably better restaurant performance in many applications than theperformance produced by the less sophisticated methods previouslymentioned.

An Exemplary Implementation

FIG. 8 individually depicts various functional elements constituting thebuffer manager system 18 according to a preferred embodiment of thepresent invention. As illustrated in FIG. 8, the major elements(including software modules) in the buffer manager system 18 include thearrival detector 120, the demand predictor 122, the production manager124, the component manager 126, the grill manager 128, and the datalogger 130. These elements cooperate with one another to implementvarious steps illustrated in FIG. 4. Because of the earlier detaileddiscussion of all the steps illustrated in FIG. 4, a detailed discussionof various modules or elements shown in FIG. 8 is not given. However, abrief description of each module is given hereinbelow for ease ofreference.

Arrival Detector

As discussed earlier hereinbefore, the arrival detector 120 analyzes alive stream of images supplied by, for example, an entrance camera 131and signals a software event when a car has entered the restaurantproperty.

Demand Predictor

The demand predictor 122 uses the car arrival events generated by thearrival detector 120 to generate an estimate of near-term demand forindividual food product components (e.g., burger patties). In apreferred embodiment, the demand predictor 122 simply counts the numberof car arrivals in the last M seconds and multiplies this count by ademand factor D to produce a projected near-term demand estimate E foreach food product component of interest. The list of these components ofinterest, as well as the individual value of M and D for each component,is retrieved from the configuration database 132. The demand predictor122 may update its estimates E once every second as well as immediatelyupon a car arrival event.

Production Manager

The production manager's 124 primary function is to coordinate theoperations of the component manager 126 and the grill manager 128.

Component Manager

The component manager 126 is responsible for tracking the actual levelsof available food product components in the staging buffer, as well ascomputing the proper amount of additional components needed to maintainan appropriate buffer level and satisfy impending demand. Tracking ofcurrent buffer levels is accomplished by monitoring the input suppliedby the kitchen crew via the assembly and grill interfaces 134, 136 andtheir respective hardware data input/display monitors 135, 137. Thegrill interface 136 informs the component manager 126 of actualproduction events (i.e., supply of new components to the buffer), whilethe assembly interface 134 signals when and how many units of aparticular component are withdrawn from the buffer to assemblesandwiches. The component manager 126 may also generate a value called“buffer deficit” for each buffered component. This value indicates thenumber of components needed, in addition to the current amount on hand.The deficit number is the sum of two components: Target Deficit andProjected Demand. The target deficit is the difference between thecurrent target level and the current actual buffer level, whileprojected demand is the output from the demand predictor 122. Thecurrent target level is obtained from a “buffer schedule” stored in theconfiguration database 132. This schedule contains the appropriatetarget levels, as determined by the human restaurant manager, forparticular time periods of the day.

Grill Manager

The grill manager 128 determines when to inform the kitchen crew in therestaurant's grill section that a batch of a particular food productcomponent should be cooked. In order to make the production decision,the grill manager 128 may consider the current buffer deficit valuecomputed by the component manager 126 as well as the “batch size” ineffect for a particular component, i.e., the number of components thatshould be cooked simultaneously. Batch sizes for each component are partof the buffer schedule so that different values may be in effect duringdifferent times of the day. In one embodiment, a production decision issignaled whenever the current buffer deficit exceeds one-half of thecurrent batch size. The grill manager 128 sends this decision event tothe grill interface 136 to be displayed on the terminal 137 for thegrill crew.

Assembly Interface

The assembly interface module 134 implements the user-interface code todrive the touch-screen 135 located in the food assembly area. Assemblycrewmembers touch button areas on the screen 135 to indicate the typeand number of sandwiches they are producing to fill customers' orders.This information is then sent to the component manager 126.

Grill Interface

The grill interface module 136 implements the user-interface code todrive the touch-screen 137 located, for example, above a grill It mayalways display the current buffer levels, as well as indicating when aproduction decision was received by the grill manager 128. In the lattercase, the desired batch size may be displayed as well. Crewmembers mayrespond by touching corresponding button areas on the screen 137 to tellthe grill manager 128 when they have started a cooking cycle. The grillinterface 136 conveys this information to the grill manager 128.

Data Logger

The data logger 130 collects operational data across the entire buffermanagement system and writes them to a database 138 for future referenceor for troubleshooting.

Manager Console

The manager's console 140 may be a back-office user interface modulethat allows the human restaurant manager to monitor the whole foodpreparation and delivery operations and adjust operational parameters(of one or more software modules in FIG. 8) as needed using thetouch-screen monitor 141.

It is noted that the discussion hereinabove focused on “deterministic”or model-based methods of detecting vehicle and/or people on arestaurant's property. For example, the buffer manager 20 according tothe present invention automatically produces a mapping from a leadingindicator of demand (e.g., car and people counts) to a time-delayedproduction order for either food product components or completed foodproducts. The foregoing discussion focused on a set of implementationsthat use so-called “deterministic” mappings—i.e., explicit models of thephysical processes that relate leading demand measurements to productionorders. However, it is noted that such explicit modeling is not requiredin order to yield the same result. Other “non-deterministic” ornon-model based methods such as, for example, neural network-basedmethods, fuzzy logic-based methods, etc., may also be used for the samepurpose. That is, a “non-deterministic” mapping may equally capture themapping.

The foregoing describes a real-time buffer manager system thatcalculates optimal food buffer levels, for both completed products andproduct components, based on real-time counts of restaurant patronsthroughout a restaurant's property. The real-time buffer manager employsa computer vision system, running a series of 2D image processingtechniques that detect and track vehicles and people in several cameraviews. The system's cameras may be pointed at any of several keylocations throughout the restaurant's property, including the property'sentrance and exits, the drive-thru lane, the parking area, therestaurant's entrance and exit doors, and the front counter area. Patroncounts are fed from the computer vision system into a queuing model thatestimates when each patron will arrive at an ordering station.Simultaneously, a parametric observer takes inputs from the kitchen crewto track several key pieces of production information including thenumber of products and components in the food buffer, and averageservice times for ordering and production. The buffer manager estimates,in real-time, the probable demand for completed food products and foodproduct components, based on the number of patrons detected and theestimated time for them to arrive at an ordering station. Thisinformation is displayed to the kitchen crew, who then can prepare therequired food products and food product components. Thus, instead ofanalyzing historical sales data, the buffer manager according to thepresent invention electronically performs direct measurement of probablefuture demand, and electronically predicts, in real-time, what thefuture food product demand will be in a predetermined time (e.g., 3-5minutes) immediately following the direct measurement of the demand.

While the invention has been described in detail and with reference tospecific embodiments thereof, it will be apparent to one skilled in theart that various changes and modifications can be made therein withoutdeparting from the spirit and scope thereof. Thus, it is intended thatthe present invention cover the modifications and variations of thisinvention provided they come within the scope of the appended claims andtheir equivalents.

1. A method of managing food supply in real time comprising:electronically generating real time data about food consumers inside orin the vicinity of a food outlet; and electronically predicting, basedon said real time data, an amount of food to be ordered by said foodconsumers in a predetermined time interval immediately following saidgeneration of said real time data.
 2. The method of claim 1, whereinelectronically generating said real time data includes at least one ofthe following: placing a first plurality of sensors inside said foodoutlet to generate, in real time, a first information about potentialfood consumers inside said food outlet; placing a second plurality ofsensors around said food outlet to generate, in real time, a secondinformation about human and automobile traffic flow external to saidfood outlet and in the vicinity thereof; and electronically analyzingsaid first and said second information in real time to generate saiddata about said food consumers.
 3. The method of claim 2, wherein atleast one of said first and said second plurality of sensors is one ofan electrical sensor; a mechanical sensor; and a chemical sensor.
 4. Themethod of claim 3, wherein said electrical sensor is one of a magneticloop sensor; a laser beam switch; and a camera.
 5. The method of claim3, wherein said mechanical sensor is one of a piezo-electric switch; anda pneumatic pressure switch.
 6. The method of claim 3, wherein saidchemical sensor is a device that measures changes in CO (carbonmonoxide) emissions.
 7. The method of claim 2, wherein said firstinformation includes indications of one or more of the following:whether said potential food consumers are present inside said foodoutlet; a total number of said potential food consumers inside said foodoutlet; a direction of motion of one or more of said potential foodconsumers; whether one or more of said potential food consumers areplacing food orders; a food ordering pattern of each of said potentialfood consumers that is placing a food order; and how long it takes foran employee of said food outlet to receive said food order.
 8. Themethod of claim 2, wherein said second information includes indicationsof one or more of the following: direction of movement of said human andsaid automobile traffic flow including: whether one or more humans insaid human traffic are entering or exiting the property of said foodoutlet, and whether one or more vehicles in said automobile traffic areentering or exiting the property of said food outlet; a total number ofhuman food consumers entering and exiting said food outlet; a totalnumber of vehicles constituting said automobile traffic; whether saidhuman and automobile traffic is present on the property of said foodoutlet; whether one or more vehicles in said automobile traffic areentering or exiting a drive-thru food order lane on the property of saidfood outlet; and a total number of vehicles entering or exiting saiddrive-thru lane.
 9. The method of claim 2, wherein said real time dataincludes one or more of the following: a first amount of time that ittakes, on average, for one of said potential food consumers to waitprior to placing a food order; a first number of potential foodconsumers inside said food outlet; a second number of vehicles on theproperty of said food outlet; a third number of automobiles in adrive-thru food order lane on the property of said food outlet; a secondamount of time that it takes, on average, for an automobile to remain insaid drive-thru lane prior to arriving at a drive-thru ordering window;a third information about a corresponding ordering pattern of each foodconsumer present inside said food outlet or in said drive-thru lane; foreach food consumer entering said food outlet, a corresponding fourthamount of time that it takes for each said human food consumer to walkto a food order panel inside said food outlet after entering theproperty of said food outlet; and for each food order received, acorresponding fifth amount of time that it takes for an employee of saidfood outlet to receive each said food order.
 10. The method of claim 2,wherein electronically analyzing said first and said second informationincludes electronically performing at least one of the following in realtime: processing said second information to identify and count vehiclesconstituting said automobile traffic external to and in the vicinity ofsaid food outlet; further processing said second information to identifyand count vehicles present in a drive-thru food order lane on theproperty of said food outlet; and processing said first and said secondinformation to identify and count humans inside and in the vicinity ofsaid food outlet.
 11. The method of claim 1, wherein electronicallypredicting said amount of food to be ordered includes electronicallyperforming the following in real time: selecting one or more queuingmodels from a plurality of queuing models; inputting relevant portionsof said real time data into respective one or more queuing modelsselected; and simulating each of said one or more queuing models afterinputting said relevant portions of said real time data thereinto. 12.The method of claim 11, where said plurality of queuing models includesa zero queue model; a single queue model; a single queue, multiplestation model; and a multi queue model.
 13. The method of claim 1,further comprising electronically performing at least one of thefollowing in real time: estimating impending food product demand in viewof said prediction of said amount of food to be ordered by said foodconsumers; and estimating demand for each completed food productavailable for consumption.
 14. The method of claim 13, whereinestimating impending food product demand includes electronicallyperforming at least one of the following: estimating a desired nominalcomponent buffer level, wherein said component buffer level includes aplurality of bins of food product components; estimating remaining timefor each of said plurality of bins before each said bin runs out ofcorresponding food product components; and estimating time required tofill one of said plurality of bins with corresponding food productcomponents when said bin becomes empty.
 15. The method of claim 13,wherein estimating impending food product demand includes displayingestimated food product demand on a display terminal.
 16. The method ofclaim 13, wherein estimating demand for each completed food productincludes estimating said demand for each said completed food productusing a food production data input received from an employee of saidfood outlet.
 17. The method of claim 1, wherein electronicallygenerating real time data includes electronically tracking objectspresent inside or in the vicinity of said food outlet to identifypresence of said food consumers and to count a number of said foodconsumers present inside or in the vicinity of said food outlet.
 18. Amethod of managing food production and delivery in real time comprising:electronically predicting, based on real time data, an amount of food tobe ordered at a food outlet in a specified time interval immediatelysucceeding a generation of said real time data; preparing said amount offood predicted to be ordered; and serving prepared food to patrons ofsaid food outlet.
 19. The method of claim 18, further comprisingelectronically performing at least one of the following in real time:estimating impending food product demand in view of said prediction ofsaid amount of food to be ordered; and estimating demand for eachcompleted food product available for consumption.
 20. A real time foodbuffer management system comprising: means for electronically generatingreal time data about food consumers inside or in the vicinity of a foodoutlet; and means for electronically predicting, based on said real timedata, an amount of food to be ordered by said food consumers in apredetermined time interval immediately following said generation ofsaid real time data.
 21. A computer-readable data storage mediumcontaining a program code, which, upon execution by a processor, causessaid processor to perform the following in real time: generate dataabout food consumers inside or in the vicinity of a food outlet; andpredict, based on said data, an amount of food to be ordered by saidfood consumers in a predetermined time interval immediately followingsaid generation of said data.
 22. The data storage medium of claim 21,wherein said program code, upon execution, causes said process tofurther perform at least one of the following in real time: estimateimpending food product demand in view of said prediction of said amountof food to be ordered; and estimate demand for each completed foodproduct available for consumption.
 23. A system for managing food supplyin real time comprising: a plurality of sensors placed inside and in thevicinity of a food outlet, wherein said plurality of sensorselectronically tracks objects present inside and in the vicinity of afood outlet to generate electrical signals containing information aboutpresence of food consumers inside and in the vicinity of said foodoutlet; and a computer containing a program code, which, upon executionby a processor in said computer, causes said processor to perform thefollowing in real time: analyze said electrical signals to generatedigital data about said food consumers, and predict, based on saiddigital data, an amount of food to be ordered by said food consumers ina predetermined time interval immediately following generation of saiddigital data.